In 20 years, the Internet has transformed from a place of information exchange to a cyberspace full of hidden folds, ecosystems, microecosystems, and autonomous creatures that traverse it in search of data.
A case in point are Bots: structured programs to retrieve information from the web. We are talking about a phenomenon so widespread that, according to research by Encapsule, since 2016 bots have generated more traffic than humans themselves.
The most well-known type of bot is that of search engines: from Google to all the other lesser-known engines, they all use bots, called spiders, to index the content of websites. Alongside these are bots that check the security of a site, looking for vulnerabilities to exploit illegally.
But the scenario is vast, you will encounter bots specialized in tasks that few people know about. First, there are those that monitor e-commerce sites on behalf of their competitors. They are very popular and continuously check the price trend to allow those who control them to be able to propose special offers, price reductions and, in general, always have an advantage over competitors.
Still in the e-commerce field, those bots dedicated to buying goods at advantageous prices are also very popular. Basically, a software keeps tabs on sites like Amazon, Yoox and the like to take advantage of special offers and buy goods as soon as the price drops.
In some cases, the program is able to immediately relist what has been purchased automatically on Ebay. The practice is so widespread that this year it has been a real bot war during Black Friday (which now lasts weeks).
All of these are bots that directly aim to ensure an economic advantage, but their continuous accesses also weigh on other aspects. The first is traffic.
If bots start hammering a site with millions of requests, everything starts to slow down and the user experience for those in the flesh becomes of poor quality, as well as increasing the consumption of web infrastructure and therefore costs.
Then there are all the implications on the site’s statistics, which will therefore no longer be “faithful” to the real traffic trend, thus generating difficulties and additional costs in promoting the business.
Another well-known issue for Web professionals is the improbable increase in cyber attacks, especially DDos. The world of gaming and e-commerce platforms are the areas most affected by DDoS attacks. This is according to research by OVH based on an analysis of the most attacked IP addresses during 2017 and the profiles of their users.
Netscout Arbor’s Active Threat Level Analysis System (ATLAS) provides an immediate snapshot of the main types of attacks in May 2018: the company collects anonymous traffic data from 400 service providers on a global scale, offering the ability to observe about one-third of all Internet traffic. As can be seen from the infographics, there were 483,910 DDoS attacks in May globally, of which 4,353 in Italy.
We have also encountered problems of both types for our customers. This is why our VMEngine Cloud Solution Architects have developed a complex system of automations, capable of shielding our customers from external attacks and unwelcome traffic.
The starting solution is provided by Amazon Web Services, a solution that, however, is very restrictive and does not allow the passage of the User-agents of the Bots fundamental for indexing and therefore for SEO, ending up blocking even the Bots of Google and other search engines, and it is also not modeled on the specific CMS (WordPress, Magento, Prestashop, etc.), thus not being perfectly secure.
The solution proposed by VMEngine aims to customize the solution as much as possible on the specific needs of each individual project, associating specific rules that allow the transit of benevolent User Agents and considering rules tailored to the main CMS of our customers, namely WordPress and Magento. This solution leverages AWS CloudFormation to quickly and easily set up AWS WAF rules that help block the following common attacks:
- SQL injection: Hackers inject malicious SQL code into web requests to extract data from the database. This solution is designed to block web requests that contain potentially malicious SQL.
- Cross-site scripting: Also known as XSS, hackers exploit vulnerabilities in a benign website as a vehicle to inject malicious scripts on the client’s site into a legitimate web browser. This solution is designed to inspect commonly explored elements of incoming requests to identify and block XSS attacks.
- HTTP Floods: Web servers and other back-end resources are at risk of distributed denial of service (DDoS) attacks, such as HTTP floods. This solution automatically triggers a rate-based rule when web requests from a client exceed a configurable threshold.
- Scanners and probes: Malicious sources scan and scan web applications with internet access for vulnerabilities. It sends a series of requests that generate 4xx HTTP error codes, and you can use this history to identify and block malicious source IP addresses. This solution creates an AWS Lambda function that automatically analyzes Amazon CloudFront or Application Load Balancer access logs, counts the number of bad requests from unique source IP addresses, and updates AWS WAF to block further scans from those addresses.
- IP reputation lists: A number of organizations maintain reputation lists of IP addresses maintained by known attackers, such as spammers, malware distributors, and botnets. This solution leverages the information contained in these reputation lists to help you block requests from malicious IP addresses.
- Bots and scrapers: Operators of publicly accessible web applications must have confidence that clients accessing their content accurately identify themselves and will use the services as intended. However, some automated clients, such as content scrapers or malicious bots, pose as deceptive to bypass restrictions. This solution allows you to identify and block faulty bots and scrapers.
AWS services are already natively prepared for DDoS mitigation, but it is their combination that can make any web infrastructure secure, efficient and with low consumption (costs). Specifically, our Solutions Architects have used the following Amazon Web Services services to create the above Custom solutions:
- Amazon CloudFront
- AWS Application Load Balancer (ALB, essential to the solution)
- AWS Web Application Firewall (WAF)
- AWS API GateWay
- AWS Lambda
- AWS CloudWatch
- AWS CloudFormation
As you can see from the table, the costs of this security solution are small, vastly lower than the benefits generated in terms of security, consumption and user experience.
The extraordinary increase in DDoS attacks and traffic generated by Bots is pushing CEOs and CTOs of all E-commerce companies, of all sizes, to make choices capable of safeguarding the business from any external threat. Also because, considering the cost table provided, a traffic of, for example, a million requests that were not blocked at the beginning, could generate a consumption cost for the infrastructure even 100 times higher than the cost of our security solution.
It is therefore necessary to adapt the infrastructure of one’s online business both to the new needs of the market and to the new dangers coming from the network. Even those who have implemented mitigation strategies must remember that they have not installed a magical apparatus that mitigates every attack. Attacks evolve and so must defenses evolve as well. It will become more complex to protect yourself from attacks.